Terms in RL Post-Training

Explain terms frequently appearing in AI articles.

Rollout, Trajectory

In the context of RL, rollout typically means generating an alternating sequence of state, action and reward. This sequence is referred to as trajectory. In the context of RL post-training of LLM, it usually means generate following tokens and obtain rewards. In the context of VLA, it means generate a sequence of arm pose, gripper pose, action. (which is literally fit components into state, action and reward respectively)

On-policy (Off-policy) training

On-policy refers to that the model is trained on trajectories generated by the model itself. While off-policy refers to that the model is trained on trajectories generated by other models or is trained on collected data, e.g., SFT.

Inverse reinforcement learning (IRL)

IRL provides a classical solution to situation where explicit reward functions are not available that

Critic or Value Function

Predicts expected future reward, and is used to compute advantage.

---